
PATENT 
450110-03711 



IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 



Applicants 
Serial No. 
Filed 
For 

Art Unit 



Jason Charles PELLY et al . 
10/010, 026 
December 6, 2001 
EMBEDDING DATA IN MATERIAL 



2621 



74 5 Fifth Avenue 

New York, New York 10151 

Tel. (212) 588-0800 



I hereby certify that this correspondence is being 
deposited with the United States Postal Service as 
first class mail in an envelope addressed to: 
Assistant Commissioner for Patents 

Washington, D.C. 20231, on February 7, 2002 

Bruno Polito, Reg. No. 3 8,580 
Name of Applicant, Assignee or 
Registered Representative 



Signature 
February 7, 2002 




Date of Signature 



CLAIM OF PRIORITY 

Assistant Commissioner for Patents 
Washington, D.C. 20231 

Sir : 

In support of the claim of priority under 35. U.S.C. 
§ 119 asserted in the Declaration accompanying the above-entitled 
application, as filed, please find enclosed herewith certified 
copies of U.K. Application Nos . 0029865.3 and 0029869.5, filed in 
U.K. on 7 December 2000 and 7 December 2000, respectively, 
forming the basis for such claim. 



J : \SONY\03711\FIRCMPPD . BP (WSF : sar ) 



-1- 



i 



PATENT 
450110-03711 

Acknowledgment of the claim of priority and of the 

receipt of said certified copy(s) is requested. 

Respectfully submitted, 

FROMMER LAWRENCE & HAUG LLP 
Attorneys for Applicants 




Reg. No. 38,580 
Tel. (212) 588-0800 

Enclosure (s) 



J : \SONY\03711\FIRCMPPD. BP (WSF : sar) 




f 



* Patent - 
% Office | 



CERfTIRED COPY OF 
PRIORITY DOCUMENT 



INVESTOR IN PEOPLE 



The Patent Office 
Concept House 
Cardiff Road 
Newport 
South Wales 
NP10 8QQ 



I, the undersigned, beingan ^£SS 
of the Deregulation and Contracting Out Ac 1994 to sign ana documents 
the Comptroller-General, hereby certify that ^^^^^^r with the 
as originally filed in connection with the patent application .defied I fcerein tog ^ 
Statement of inventorship and of right to grant of a Patent (Form 
subsequently filed. 

In accordance with the Patents (Companies ^^^^^^ 
in this certificate and any -^^^^^-^S immediately before re- 
1980 with the same name as that with ^.^J^^^^^^ 
registration save for the substitution as, or f?-^^^"^ of the company 
"public limited company" or their equivalents m ^^^^fy^ to the name 
in this certificate and any accompanying documents shall be treated as 
with which it is so re-registered. 

,„ accordance with the rules, ft. words -pibrfc MM company- may be replaced b, p.l.c, 
pic, P.L.C. or PLC. 

Re-registration under the Companies Act does not constitute a new legal entity but merely 
subjects the company to certain additional company law rules. 




Signed 
Dated 3 0 NOV 2004 



Patents Form 1/77 

Patents Act 1977 
(Rule 16) 



^ x Request for a grant of a patent 

f 

s 

(See the notes on the back of this foon you can also get an 
explanatory leaflet from the Patent Office to help you fill in 
this form) 




858:00 £589717-53 VOmt 
01/7700 0,00-0029865,3 



&CO 



The Patent Office 

Cardiff Road 
Newport 
Gwent NP9 1RH 



1. Your reference 


P010031GB RWP 




2. Patent application number 

(The'PatentOWce'willfiilin-this-part) 


^7 BK 2 000 | 


0029865.3 



3. Full name, address and postcode of the 
or. of each applicant 
(underline all surnames) 



Sony United Kingdom Limited 

The Heights 

Brooklands 

Weybridge 

KT13 0XW 

United Kingdom 



Patents ADP number (if you know it) 



If the applicant is a corporate body, give 
the country/state of its incorporation 



UNITED KINGDOM 



4. Title of the invention 



Embedding Data in Material 



5. Name of your agent (if you have one) 

"Address for service" in the United Kingdom 
to which ail correspondence should be sent 
(including the postcode) 

Patents ADP number (if you know it) 



D YOUNG & CO 

21 NEW FETTER LANE 

LONDON 

EC4A 1DA 

59006 



If you are declaring priority from 
one or more earlier patent 
applications, .give the country and 
date of filing of the or each of these 
earlier applications and (if you know 
it) the or each application number 



1st 



Country Priority application Date of filing 
number (day/month/year) 
(if you know it) 



2nd 
3rd 



7. If this application is divided or otherwise Number of earlier Date of filing 

derived from an earlier UK application, application (day/month/year) 

give the number and filing date of the 

earlier application 



Is a statement of inventorship and of right to grant of a patent 
required in support of this request? (Answer 'Yes' if: 

a) any applicant named in part 3 is not an inventor, or - 

b) there is an inventor who is not named as an applicant, or 

c) any named applicant is a corporate body. 
See note (d)) 



Enter the number of sheets 
for any of the following items 
you are filing with this form. 
Do not count copies of the 
same document 



Continuation sheets of this form 

Description 

Claim(s) 



Abstract 
Drawing(s) 



YES 



r 



1 0. If you are also filing any of the 
following, state how many 
against each item 



Priority Documents 



Translation of Priority Documents 



Statement of inventorship and right to 
grant of a patent (Patents Form 7/77) 



Request for preliminary examination 

and search (Patents Form 9/77) 



Request for substantive 
examination (Patents Form 1CV77) 



Any Other documents (Please specify) 



11. 



I/We request the grant of a Patent on the basis of this application. 



Signature 

D YOUNG & CO 

Agents for the Applicants 



Date 



7 Dec 2000 



12. Name and daytime telephone number of person 
to contact in the United Kingdom 



Richard Pratt 



023 80719500 



Warning 

After an application for a patent has been filed, the Comptroller of the Patent Office will consider whether publication or 
communication of the invention should be prohibited or restricted under Section 22 of the Patents Act 1977. You will be 
informed if it is necessary to prohibit or restrict your invention in this way. Furthermore, if you live in the United Kingdom, 
Section 23 of the Patents Act 1977 stops you from applying for a patent abroad without first getting written permission from the 
Patent Office unless an application has been filed at least 6 weeks beforehand in the United Kingdom for a patent for the same 
invention and either no direction prohibiting publication or communication has been given, or any such direction has been 
revoked. 

Notes 

a) If you need help to fill in this form or you have any questions, please contactthe Patent Office.on 01645 500505. 

b) Write your answers in capital letters using black ink or you may type them. 

c) if there is not enough space for all the relevant details on any part of this form, please continue on a separate sheet of 
paper and write "see continuation sheet" in the relevant part(s) Any continuation sheets should be attached to this form. 

d) If you answered 'Yes' Patents Form 7/77 will need to be filed. 

e) Once you have filled in the form you must remember to sign and date it. 

f) For details of the fee and ways to pay please contact the Patent Office. 



Patents Form 7/77 

. )nts Act 1977 
(Rule 15) 



Statement of inventorship and 
of right to grant of a patent 





The Patent Office 

Cardiff Road 
Newport 

Gwent NP9 1RH 



1. Your reference 



P01 0031GB RWP 



2. Patent application number (if you know it) 



3. Full name of the or of each applicant 



SONY UNITED KINGDOM LIMITED 



4. Title of the invention 



EMBEDDING DATA IN MATERIAL 



5. State how the applicant^; derived the right from the BY VIRTUE OF AN ASSIGNMENT DATED 14 
inventory) to be granted a patent DECEMBER 2000. 



6. How many, if any, additional Patents Forms 7/77 are 
attached to this form? (see note (c)) 



7. 



I/We believe that the personfs) named over the page 
(and on any extra copies of this forms) is/are the 
inventory of the invention which the above patent 
relates to. 



Signature 



D YOUNG & CO 

Agents for the Applicants 



Date 



15 Dec 2000 



8. Name and daytime telephone number of person 
to contact in the United Kingdom 



023 80634816 



Richard Pratt 



Notes 

a) If you need help to fill in this form or you have any questions, please contact the Patent Office on 0645 500505. 

b) Write answers in capita} letters using black ink or you may type them. 

c) If there are more than three inventor, please write the names and addresses of the other inventors on the back of another Patents Form 7/77 and attach it to 
this form. 

d) When an application does not declare any priority, or declares priority from an earlier UK appBcation, you must provide enough copies of this form so that the 
Patent Office can send one to each inventor who is not an applicant. 



e) Once you have filled in the form you must remember to sign and date it 



Page -7 



73r > r^V 



D Young & Co ref: P01 0031GB RWP 



Enter the full names, addresses and 
postcodes of the inventors in the boxes 
and underline the surnames 



Surname PELLY 



First Names Jason Charles 

Address 2 Odell Close 
Lower Earley 
Reading 
Berkshire 
RG6 4DU 
United Kingdom 



Patents ADP number (if you know it): 



Surname TAPSON 



First Names Daniel 
Address 



3 Queen Mary Avenue 
Basingstoke 
Hampshire 
RG21 3PL 
United Kingdom 



Patents ADP number (if you know it): 





Surname KEATING 






First Names Stephen Mark 

Address 28 Huntingdon Close 
Lower Earley 
Reading 
Berkshire 
RG6 3AB 
United Kingdom 




Patents ADP number (if you know ^^Q^j ^J^^l^^Ooj 



Have you signed the form? 



Page*2 



DUPLICATE 

1-00-81, P/1003IGB 1 

Embedding Data in Material 

The present invention relates to embedding data in material. 
In this specification "material 56 means any one or more of video material, 
audio material and data material. 'Video' is generic to still and moving images. 
5 Steganography 

Steganography is the embedding of data into material such as video material, 
audio material and data material in such a way that the data is imperceptible in the 
material. 



Data may be embedded as a watermark in material such as video material, 
10 audio material and data material. A watermark may be imperceptible or perceptible in 
the material. 

A watermark may be used for various purposes. It is known to use watermarks 
for the purpose of protecting the material against, or trace, infringement of the 
intellectual property rights of the owner(s) of the material. For example a watermark 
1 5 may identify the owner of the material. 

Watermarks may be "robust" in that they are difficult to remove from the 
material. Robust watermarks are useful to trace the provenance of material which is 
processed in some way either in an attempt to remove the mark or to effect legitimate 
processing such as video editing or compression for storage and/or transmission. 
20 Watermarks may be "fragile" in that they are easily damaged by processing which is 
useful to detect attempts to remove the mark or process the material. 

Visible watermarks are useful to allow e.g. a customer to view an image e,g. 
over the Internet to determine whether they wish to buy it but without allowing the 
customer access to the unmarked image they would buy. The watermark degrades the 
25 image and the mark is preferably not removable by the customer. Visible watermarks 
are also used to determine the provenance of the material into which they are 
embedded. 

It is known to embed data in material. It is desirable to do that and allow the 
data to be removed from the material to avoid degrading the material. It is known to 
30 combine the data with the material, the data being scaled by a scaling factor which is 
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chosen according to desired properties of the data when combined with the material. 
Those properties include one or more of: concealing the data in the material; making 
the data perceptible in the material; making the data, when embedded in the material, 
resistant to processing which, intentionally and unintentionally, removes or damages 
the embedded data. 

According to a first aspect of the invention, there is provided a method of 
embedding data in material, the method comprising 

combining a representation of the material with a function of the data and a 
scaling factor; wherein 

the scaling factor is generated as a function of a trial decoding of the material, 
the trial decoding comprising processing the material to recover data therefrom. 
Thus the scaling factor can be chosen on the basis of an estimate of the result 
of a process (e.g. decoding) which will be performed on the combined material and 
data in practice so as to increase the likelihood that the data is recoverable from the 
15 material. 

An embodiment of the invention further comprises the steps of: 
combining, as a trial, a representation of the material with a function of the data and a 
trial scaling factor; and 

performing, as a trial, a predetermined process on the combined material and data; 
20 wherein the scaling factor is generated as a function of a trial decoding of the 

processed combined material and data. 

Thus the scaling factor can be chosen on the basis of an estimate of the result 
of a process (e.g. JPEG processing) which could be performed on the combined 
material and data in practice and which may damage the embedded data, so as to 
25 increase the likelihood that the data will not be damaged by such processing . 

According to a further aspect of the invention, there is provided a method of 
embedding data in material, the method comprising the steps of: 

producing transform coefficients Ci representing a transform of the material; 
producing a pseudo random symbol sequence (PRSS) having L symbols Pi of 
30 values +1 and -1; 

calculating the correlation S=Z Ci.Pi, for i=l to i=L; and 



) 
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) calculating modified coefficient values Ci' = Ci + a * Pi, where a is calculated 

dependent on S and the value of the data bit to be embedded in the coefficient. 

Preferably 

5 a= (a'+ offset) 

where a' + offset is a function of the data bit to be embedded in the coefficient, 
and the method comprises the step of calculating modified coefficient values 
Ci' = Ci + (<x'+ offset)*Pi where 
a' =0 if S is positive and the data to be concealed is a bit of a first value, 

10 a' =0 if S is ne gative and the data to be concealed is a bit of a second value, 

and otherwise a' is a function of S such that Z Ci'.Pi has the correct sign to 
represent the bit to be encoded. 

It will be noted that the calculation of the correlation S=S Ci.Pi, for i=l to i=L 
is a form of trial decoding as in the said first aspect and the scaling factor is chosen in 
1 5 dependence on that correlation. 

A further aspect of the invention provides a computer program product 
arranged to carry out one of the aforesaid methods when run on a computer. 

The invention also provides corresponding apparatus in other aspects of the 
invention. 

20 For a better understanding of the present invention, reference will now be made 

by way of example to the accompanying drawings in which: 

' • Figure 1 is a schematic block diagram of an embodiment of a watermarking 
system according to the present invention; 

Figure 2 is a schematic block diagram of another embodiment of a 
25 watermarking system according to the present invention; 

Figure 3 A is a schematic diagram of a wavelet transform showing the 
relationship of the symbols of a pseudo random symbol sequence to coefficients ; 

Figure 3B is a flow diagram of calculations performed by the system of Figure 

2; 

30 Figure 4 is a schematic block diagram of an illustrative watermark decoding 

and removal system; 



1-00-8 1, P/ 1 003 1GB 



4 



") Figure 5 is a flow diagram of calculations performed by the watermark 

remover and decoder of Figure 4; 

Figure 6 is a schematic block diagram of a further embodiment of a 
watermarking system according to the present invention; 
5 Figures 7 to 10A are schematic block diagram of subsystems of the system of 

Figure 6; 

Figures 10B and 10C are flow diagrams illustrating a process for calculating a; 
Figures 1 1 and 12 are diagrams explaining wavelet transforms; and 
Figures 13 and 14 are diagrams showing a UMID and a data reduced UMID. 
10 Overview 

Referring to Figure 1, in the shown illustrative watermarking system, a spatial 
domain image I produced by a source 1 is combined with watermark data Ri to 
produce a spatial domain watermarked image I'. The watermarked image is produced 
by an embedder 3 according to the equation 
15 Ci'=Ci + cc. Ri 

where Ci and Ci' are, for example, wavelet transform coefficients of the image, 
and a is a scaling factor which is also referred to herein as 'strength', a is chosen in 
this example so that the watermark is imperceptible in the image and to resist removal 
of the watermark by unauthorised processing. It is thus desirable that a has the 
20 smallest value which achieves that. If a is too big the watermark is perceptible in the 
image; if it is too small the mark may not survive processing of the image. 

-.In accordance with this embodiment, a is determined from a trial decoding of 
the original unmarked image I in a decoder 4. The decoding is that which would be 
used to decode the watermarked image I\ A value a' is produced by a calculator S3- 
25 S8, to which an offset value is added by an adder S9 to produce a. This produces 
values of a over the image, which are used to scale the data Ri so as to conceal the 
data. 

Figure 2 shows an example of the system of Figure 1 in which the calculation 
of a also encodes Ri. Figure 6 shows an example of the system of Figure 1 in which 
30 the trial decoding takes place after a trial processing of the image data by a process 
which is likely to be applied to the watermarked image. 
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The data Ri in the examples given below is derived from a UMID. UMIDs are 
described in the section UMIDs below. As mentioned above, and in the examples 
given below, the image is subjected to a wavelet transform. Wavelet transforms are 
briefly discussed in the section Wavelets below. 
5 Figure 2 

Referring to Figure 2, for the purposes of explanation, it is assumed that the 
wavelet transform applied to the original spatial domain image results in a transform 
having four sub-bands of level 1: see the section Wavelets below. For ease of 
explanation the following description will refer only to the upper horizontal sub-band, 
10 but it will be appreciated that the present invention can be applied to any sub-band and 
may be applied to a plurality of sub-bands. The coefficients of the wavelet transform 
are denoted by Ci where i is the i* coefficient of a sequence of JxL coefficients where 
there are J bits of UMID data Wl to WJ. As will become apparent the JxL coefficients 
correspond to symbols Pi of a Pseudo Random Symbol Sequence PRSS. Each UMID 
1 5 bit is embedded in L wavelet coefficients. 

In this example a UMID is produced as the watermark data by a UMID 
generator 6. In this example the watermark is imperceptible. The data of the UMID is 
combined with the wavelet coefficients Ci in an embedder 3 in the manner described 
in detail in the following text. The transformed image together with the watermark Ci' 
20 is subjected to an inverse wavelet transform T~ l , (5) to produce a spatial domain 
' watermarked image I 1 . 

The watermark is decoded and, optionally, removed from the watermarked 
image using the illustrative decoding and removal system of Figure 4. 
Trial decoder 4 

25 A trial decoder 4 comprises a generator 4^ a converter 4 2 , and a correlator SI . 

The generator 4\ produces a pseudo random binary sequence which is pseudo random 
because the sequence whilst appearing random can be reliably reproduced. The binary 
sequence has a length of JxL bits. The converter 4 2 converts the binary 1 and 0 to +1 
and -1 respectively to produce a pseudo random symbol sequence (PRSS) Pi of values 

30 +1 and -1. The symbols of the PRSS are denoted herein by Pi, where i denotes the 1 th 
symbol of the sequence which is JxL symbols along. The correlator produces 
correlation values 
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Sj = ICLPi, where the sum is taken over the range i = G'OL + 1 to jL for each 
of j=l to J. 

Calculation of ai and embedding the UMID 
There is one strength value aj for each UMID bit Wj. 

The calculation of aj and the embedding of the UMID in the image operates in 
accordance with the flow diagram of Figure 3B. The embedder 3 calculates a function 
Ci' = Ci + aj Pi for i = (j-l)L + 1 to jL for each of j=l to J. 

where Ci' is an ith wavelet coefficient modified to encoded a bit of 

watermarking data: and 

aj is the scaling factor, the value of which depends on: 

a) the value 1 or 0 of a bit Wj of the UMID to be encoded in modified 
coefficient Ci'; and 

b) the sign of the correlation value Sj = ECi.Pi, for i = (j-l)L + 1 to jL for each 
15 of j=l to J, produced by the trial decoder 4; and 

' c) the offset value, which is +/-1 in this example, so aj = aj' + offsetj. 
The principle of operation is that a watermark bit Wj=l is encoded as a 
positive correlation value and Wj=0 is encoded as a negative correlation value ( or vice 
versa), aj is chosen to ensure the value of a correlation Sj' = ZCi'.Pi for i = (j-l)L + 1 
20 to jL for each of j= 1 to J, performed at the decoder has the correct sign to represent the 
value of bit Wj. If the correlation Sj performed at the encoder has the correct sign, 
then c£j' = 0 otherwise aj' is modified to ensure that the correlation Sj' = ECi'.Pi 
performed at the decoder has the correct sign. 
Thus referring to Figures 2 and 3:- 
25 A value aj'=aj- offsetj. 

Step SI (correlator SI of Figure 2) calculates the correlation value Sj = ECi.Pi, 
where the sum is taken over the range i = (j-l)L + 1 to jL for a sequence i = (j-l)L + 1 
to jL of coefficients Ci and PRSS bits Pi. (Note that 'symbols' Pi have values +1 and -1 
to ensure that bits of value 0 produced by the generator 4i contribute to the value of 
30 Sj.) Step SI is a trial decoding with a trial value of aj = 0. 



% 

) 



5 



10 



1-00-31, P/10031GB 



7 



Step S2 determines whether the bit Wj of the UMID generated by generator 6 
is 1 or 0. It will be appreciated that the bit Wj is in effect encoded over L coefficients. 
If Wj =1 then steps S3 to S5 and S9 + are followed. These steps are implemented by 
blocks S3-S8 and S9 in figure 2. 

Step S3 determines the sign of the correlation Sj. If the sign is positive and the 
bit Wj is 1 then 

at step S4 aj 5 =0. 

If the sign determined at step S3 is negative but the bit Wj =1 (which should be 
encoded by Sj positive), then 

at step S5 aj'=-Sj/(L-l). 

At step S9 + , the offset +1 is added to ensure that aj is positive if Sj=0 and to 
increase robustness.. It should be noted that the offset is a signed value (+1) in this 
example. 

If Wj=0 then steps S6 to S8 and S9~ are followed. These steps are implemented 
15 by blocks S3-S8 and S9 in figure 2. 

Step S6 determines the sign of the correlation Sj. If the sign is negative and the 
bit Wj is 0 then 

at step S7 aj' =0. 

If the sign determined at step S6 is positive but the bit Wj =0 (which should be 
20 encoded by S negative), then 

at step S8aj'=-Sj/(L-1). 
• -.At step S9" the offset -1 is added to ensure that aj is negative if Sj=0 and to 
increase robustness. It should be noted that the offset is a signal value (-1) in this 
example. 

25 At step S10 the value Ci' = Ci + aj Pi is calculated for i = Q-1)L + 1 to jL . 

The value aj'=-Sj/(L-l) is an example. The value aj' could be aj'—Sj/L as 
another example. 

Watermark Decoding and Removing System ( Figures 4 and 5) 

Referring to Figure 4, the watermark removing and decoding system has an 

30 input for receiving a spatial domain watermarked image V from the system of Figure 1 . 



) 
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) The image may have been subject to image processing (not shown) between 

production by the system of Figure 1 and the receipt by the system of Figure 4. 

The received image is transformed by a wavelet transformer 46 (T) to produce 
wavelet coefficients Ci\ The coefficients Ci' are provided to a synchroniser 8 which 
5 correlates the coefficients with a PRSS generated by a generator 10. The synchroniser 
8 and the PRSS 10 carry out, in known manner, correlations with differing shifts of the 
PRSS relative to the coefficients to determine the position in the watermarked 
transformed image of the original PRSS produced at the watermarking system of 
Figure 1. Once synchronisation has been achieved the coefficients Ci' are correlated 
10 with the PRSS in another correlator 12 to produce the correlation value 
Sj'=SCi'.Pi for i = Q-1)L + 1 to jL for each of j=l to J. 



where Pi has values +1 and -1. 

The correlation value Sj' is provided to a decoder 14 and to a remover 16, the 
1 5 operations of which will be described with reference to the flow diagram of Figure 5. 
The decoder 14 extracts the UMID from the image. The watermark is removed by the 
remover 16. The resulting restored transformed image is subject to an inverse wavelet 
transform (T" 1 ) in an inverse transformer 18. 

Referring to Figure 5, the synchronisation of the PRSS with the received 
20 transformed image occurs at step S12. At step S14, the correlation value 
Sj'= ECi'.Pi for i = G-1)L + 1 to jL. 

is calculated over a length L of the PRSS. 

At step SI 6, the sign of the value Sj' is determined. If Sj' is negative then the 
25 bit of the watermark, (the UMID in this example), is 0. If Sj ' is positive the bit of the 
watermark is 1 . 

At step SI 8, 

<xj=Sj'/(L-l) 

is calculated from Sj'. (This calculation may be an approximation because it 
30 assumes that ZCi.Pi =0) 

At step S22, Ci = Ci' - aj Pi is calculated for . i = Q-l)L + 1 to jL. 
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If, in the embedding process, ccj is calculated as --y at the step S5 or S8, then 

in the decoding process aj 5 is calculated as Sj'/L at step SI 8. 
Modifications. 
5 aV Threshold on the values of Ci 

In a modification of the embodiment described above, the values of the 
coefficients Ci are compared with a threshold value The at the embedder of Figure 1 , 
and the values of the coefficients Ci' are compared with a threshold value Th d . at the 
remover 16 of Figure 4 and also at the decoder 14 of Figure 4. If the value of a 
To coefficient exceeds the threshold, that coefficient is not used in establishing the ~~ 
correlation value Sj or Sj'. The and Thd may be equal, but it has been found that Thd is 
preferably greater than The . 

By way of a simple example, assume that the PRSS has length L=4 and 
symbols PI to P4 have values +1, -1,-1, and +1. Then referring to Table 1 three 
15 examples are shown. 







PI 


P2 


P3 


P4 


Sj, aj' 






CI 


C2 


C3 


C4 


Wj=0 


Pi 


+1 


-1 


-1 


+1 




Exl 


Ci 


-2 


-5 


+1 


-3 


-1,0 


Ex2 


Ci 


-2 


-25 


+1 


-3 


+19, -6.3 


Ex3 


Ci 


-2 




+1 


-3 


-6,0 



Table 1 

Example 1 fExl) 

The coefficients Ci have values shown. If the value of the bit Wj of the 



watermark to be encoded is 0 then according to Figure 3, Sj=-1 and so aj 1 -0. 
20 Example 2 (Ex2) 

However if as shown in example 2 the coefficient C2 has a value -25 than 
Sj=+19 and aj — 6.3. Large values of aj' may cause the watermark to be perceptible 
when it should be imperceptible. 

Example 3 (EX3) 
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) In accordance with an embodiment of the present invention, a threshold Th* is 

set. It may be about 6 for the above example. In practice it is set empirically. Thus as 
shown in Table 1, the coefficient C2 is not used in the calculation of Sj, and also the 
corresponding bit of the PRSS is also not used. As a result Sj=-6 and aj'=0. 
5 b) Clipping coefficient values . 

In an alternative modification, the values of the modified coefficients Ci' are 
clipped at the decoder of Figure 4 if they exceed a threshold value Th<j. Thus 
coefficient values greater than the threshold are reduced to a predetermined value e.g. 
Thd. For example referring to Table 1 Example 3, the coefficient C2 is clipped to say 
1 0 Th d =-6 at the decoder. 

Such clipping is also performed at the embedder of Figure 2. 
Limiting oc ? 

The value of a' may be limited to be within a present range determined by 
upper and lower bounds. 
15 Trial processing and decoding- Figure 6 

Referring to Figure 6, an unmarked spatial domain Image I is applied to an 
embedder 60. An example of the embedder is shown in Figure 8. The embedder 
calculates 

Ci" = Ci + ct Tj Ri for i = (j-l)L + 1 to jL for each of j=l to J. 

20 

where: Ci is a wavelet transform coefficient of the image; Ri is a 
watermarking symbol formed by combining a PRSS of JxL bits Pi with watermark 
data Wj. Symbol Ri has a value + or - 1 ; cctj is a trial value of the scaling factor for 
UMID ( or watermark ) data bit Wj. In this example a t j is initialised to 1 . Figure 7 
25 shows an example of a subsystem, for producing Ri. 

The embedder also includes an inverse transformer which produces a spatial 
domain watermarked image Iw. 

The image Iw is processed by a processor 62 to produce a processed spatial 
domain image Ip. The processor 62 is chosen to process the image according to a 
30 process which the watermarked image is likely to encounter in use, and/or a process 
which may be applied to the image to deliberately remove or damage the watermark. 
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^ JPEG processing using DCT transforms is a process which is known to be potentially 

damaging to some watermarks. 

A decoder 64, an example of which is shown in Figure 9, decodes the 
processed image Ip. The decoder may extract the watermark data Wj. The decoder 
5 produces correlation values 

Sipj'= ZCip.Pi for i = (j-l)L + 1 to jL for each of j=l to J. 

where Cip are wavelet coefficients of the processed image Ip and the sum is 
calculated over a length L of a PRSS having JxL bits Pi. 
1 0 A calculator 66 calculates a new value of aj based on the magnitude of Sip ,j to 

produce new trial values of a t + n , j which is used as a new value in the trial embedder 
60. n is the number of iterations used to calculate a final value a t + n j which is applied 
to an embedder 69. n = 0,1,2. Several iterations may be used. Preferably the number 
of iterations is limited to a predetermined number, e.g. 4, because the process 62 may 
1 5 be non-linear ( JPEG processing is non-linear) and it is then unlikely that the iterations 
will converge to steady values of cctj. 

Examples of (a) the calculator 66 and (b) the embedder 69 are shown in 
Figures 10 and 8 respectively. 

Calculating Ri. Figure 7. 
20 A PRSS generator 71 produces a PRSS having JxL bits Pi. A UMID generator 

72 produces a UMID having bits Wj. In a modulator 73, each bit Wj of the UMID 
modulates, and is thus spread over, an L bit sequence of the PRSS. A data converter 
74 converts the binary values 1 and 0 at the output of the modulator to produce 
symbols Ri of value +1 and -1 respectively. 
25 Embedder 60 and 69.- Figure 8. 

The embedder of Figure 8 comprises a wavelet transformer 82 which produces 
the wavelet coefficients Ci and an inverse transformer 85. A multiplier 84 calculates 
a t + n j.Ri. An adder 83 adds a t + n j.Ri to Ci to produce. 

Ci' = Ci + ctt+nj.Ri for i = Q-l)L + 1 to jL for each of j=l to J. 

30 
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) Thus each coefficient Ci is modified by a value of a associated with that 

coefficient and by one symbol Ri. 

Unlike the example of Figures 2 and 3, Ri is a symbol stream comprising the 
PRSS modulated by the data to be embedded, and a is an unsigned magnitude. 
5 Decoder 64-Figure 9 

The decoder has a wavelet transformer 91 which produces wavelet coefficients 
Cip from the processed image Ip. A synchroniser 92 operating in known manner shifts 
the phase of the PRSS produced by a PRSS generator 93 so that it is in phase with the 
PRSS in the image Ip. A data converter 94 converts the PRSS values Pi to +1 and -1. 
10 A correlator calculates a correlation value 

Sipj= SCip.Pi i = G-1)L + 1 to jL for each of j=l to J. 

A decoder 96 determines the values of the data bits Wj from the sign of the 
correlation values Sip j . 
15 Calculating a-Figures 10A. 10B and IPC 

Referring to Figure 10A, new values of cc t +ij are calculated by adding an offset 
to a basic fixed value ag in an adder 99. The offsets are produced by an offset 
generator 95. The generator responds to an offset control value produced by a 
processor 97. The processor controls the offset and thus the values of cc t +ij in 
20 dependence on the correlation values Sipj. 

Figures 10B and 10C illustrate examples of the operation of the processor. 
• -. Referring to Figure 10B, the correlation values Sipj are compared at step S30 
with the corresponding symbols Wj. The correlation values Sipj are positive and 
negative, a positive value indicates a symbol 1 and a negative value a symbol 0, (if the 
25 values Sip are unchanged by the processing in processor 62).. If the signs of. Sipj 

correctly represent Wj then the magnitude of Sipj is compared with an upper threshold 
Th. If | Sipj I >Th then the value of aj is reduced for the next iteration a t +ij. If 
I Sipj | is not greater than the threshold aj either remains unchanged for the next 
iteration a t +ij or is used as the final value of atj. 
30 If the sign of Sipj indicates the incorrect value for Wj, then aj is increased for 

the next iteration a t +g. 
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Referring to Figure 10C, at step S40 a value (Sipj)/L is calculated from Sipj. 
That is the average correlation value over L symbols. That value is used as a t +i j for 
the next iteration. Preferably ct t +i j is compared with an upper threshold Th at step S42. 
If ott+ij exceeds Th, then a t +i j is reduced. Otherwise it is compared (S46) with a lower 
5 threshold TL. If a t +ij is less than TL, a t +ij is increased (S48) otherwise it is 
unchanged (S49). 

Modifications. 

Other transforms 

Whilst the invention has been described by way of example with reference to 

1 0 Wavelet transforms, it may be used with other transforms for example DCT. 

Other material 

Whilst the invention has been described by way of example with reference to 
material comprising images, e.g. video material , it may be applied to other material, 
for example audio material and data material. 
15 Other Watermark data . 

Whilst the invention has been described by way of example with reference to 
UMIDs as the watermark data, it may be used with other data as the watermark. 
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Wavelets 

Wavelets are well known and are described in for example " A Really 
Friendly Guide to Wavelets " by C Valens, 1999 and available at 
http://perso.wanadoo.fr/polyvalens/clemens/wavelets/wavelets.html. 

Valens shows that the discrete wavelet transform can be implemented as an 
iterated filter bank as used in sub-band coding, with scaling of the image by a factor 
of 2 at each iteration. 

Thus referring to Figure 12, a spatial domain image is applied to a set of high 
pass HP and low pass LP filters. At level 1, the first stage of filtering, the image is 
filtered horizontally and vertically and, in each direction, scaled down by a factor of 2. 
In level 2, the low pass image from level 1 is filtered and scaled in the same way as in 
level 1 . The filtering and scaling may be repeated in subsequent levels 3 onwards. 

The result is shown schematically in Figure 1 1 . Figure 1 1 is a representation 
15 normal in the art. At level one the image is spatially filtered into four bands: the lower 
horizontal and vertical band, lHi, lVi; the upper horizontal band hHi, lVi; the upper 
vertical band lHi, hVi; and the upper horizontal and vertical band, hHi, hVi. At level 
2, the lower horizontal and vertical band, lHi, lVi is filtered and scaled into the lower 
horizontal and vertical, band, 1H 2 , 1V 2 ; the upper horizontal band hH 2 , 1V 2 ; the upper 
20 vertical band 1H 2 , hV 2 ; and the upper horizontal and vertical band, hH 2 , hV 2 . At level 3 
(not shown in Figure 11), the lower horizontal and vertical band, 1H 2 , 1V 2 is further 
filtered and scaled. 



5 
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} UMIDs 

The UMID or Unique Material Identifier is described in SMPTE Journal March 
2000. Referring to Figure 13, an extended UMID is shown. It comprises a first set of 
32 bytes of basic UMID and a second set of 32 bytes of signature metadata. 
5 The first set of 32 bytes is the basic UMID. The components are: 

•A 12-byte Universal Label to identify this as a SMPTE UMID. It defines the 
type of material which the UMID identifies and also defines the methods by which the 
globally unique Material and locally unique Instance numbers are created. 

•A 1 -byte length value to define the length of the remaining part of the UMID. 
10 •A 3-byte Instance number which is used to distinguish between different 

'instances' of material with the same Material number. 

•A 16-byte Material number which is used to identify each clip. Each Material 
number is the same for related instances of the same material. 

The second set of 32 bytes of the signature metadata as a set of packed 
15 metadata items used to create an extended UMID. The extended UMID comprises the 
basic UMID followed immediately by signature metadata which comprises: 

•An 8-byte time/date code identifying the time and date of the Content Unit 
creation. 

•A 12-byte value which defines the spatial co-ordinates at the time of Content 
20 Unit creation. 

•3 groups of 4-byte codes which register the country, organisation and user 

codes. . 

Each component of the basic and extended UMIDs will now be defined in turn. 



% 
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The 12-byte Universal Label 

The first 12 bytes of the UMID provide identification of the UMID by the 
registered string value defined in table 1 . 



Byte No. 


Description 


Value (hex) 


1 


Object Identifier 


06h 


2 


Label size 


OCh 


3 


Designation: ISO 


2Bh 


4 


Designation: SMPTE 


34h 


5 


Registry: Dictionaries 


Olh 


6 


Registry: Metadata Dictionaries 


Olh 


7 


Standard: Dictionary Number 


Olh 


8 


Version number 


Olh 


9 


Class: Identification and location 


Olh 


10 


Sub-class: Globally Unique Identifiers 


Olh 


11 


Type: UMID (Picture, Audio, Data, Group) 


01, 02, 03, 04h 


12 


Type: Number creation method 


XXh 



10 
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Table 1: Specification of the UMID Universal Label 
The hex values in table 1 may be changed: the values given are examples. 
Also the bytes 1-12 may have designations other than those shown by way of example 
in the- table. Referring to the Table 1, in the example shown byte 4 indicates that bytes 
5-12 relate to a data format agreed by SMPTE. Byte 5 indicates that bytes 6 to 10 
relate to "dictionary" data. Byte 6 indicates that such data is "metadata" defined by 
bytes 7 to 10. Byte 7 indicates the part of the dictionary containing metadata defined 
by bytes 9 and 10. Byte 10 indicates the version of the dictionary. Byte 9 indicates 
the class of data and Byte 10 indicates a particular item in the class. 

In the present embodiment bytes 1 to 10 have fixed preassigned values. Byte 
1 1 is variable. Thus referring to Figure 14, and to Table 1 above, it will be noted that 
the bytes 1 to 10 of the label of the UMID are fixed. Therefore they may be replaced 
by a 1 byte 'Type' code T representing the bytes 1 to 10. The type code T is followed 
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) by a length code L. That is followed by 2 bytes, one of which is byte 1 1 of Table 1 

and the other of which is byte 12 of Table 1, an instance number (3 bytes) and a 
material number (16 bytes). Optionally, the material number may be followed by the 
signature metadata of the extended UMID and/or other metadata. 
5 The UMID type (byte 1 1) has 4 separate values to identify each of 4 different 

data types as follows: 

'Olh' = UMID for Picture material 
4 02h' = UMID for Audio material 
4 03h' = UMID for Data material 
10 c 04h' = UMID for Group material (i.e. a combination of related essence). 

The last (12th) byte of the 12 byte label identities the methods by which tfce 
material and instance numbers are created. This byte is divided into top and bottom 
nibbles where the top nibble defines the method of Material number creation and the 
bottom nibble defines the method of Instance number creation. 
15 Length 

The Length is a 1-byte number with the value c 13h' for basic UMIDs and 4 33h' 
for extended UMIDs. 

Instance Number 

The Instance number is a unique 3 -byte number which is created by one of 
20 several means defined by the standard. It provides the link between a particular 
'instance' of a clip and externally associated metadata. Without this instance number, 
all material could be linked to any instance of the material and its associated metadata. 

The creation of a new clip requires the creation of a new Material number 
together with a zero Instance number. Therefore, a non-zero Instance number 
25 indicates that the associated clip is not the source material. An Instance number is 
primarily used to identify associated metadata related to any particular instance of a 
clip. 

Material Number 

The 16-byte Material number is a non-zero number created by one of several 
30 means identified in the standard. The number is dependent on a 6-byte registered port 
ID number, time and a random number generator. 
Signature Metadata 
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j Any component from the signature metadata may be null-filled where no 

meaningful value can be entered. Any null-filled component is wholly null-filled to 
clearly indicate a downstream decoder that the component is not valid. 
The Time-Date Format 
5 The date-time format is 8 bytes where the first 4 bytes are a UTC (Universal 

Time Code) based time component. The time is defined either by an AES3 32-bit 
audio sample clock or SMPTE 12M depending on the essence type. 

The second 4 bytes define the date based on the Modified Julian Data (MJD) as 
defined in SMPTE 309M. This counts up to 999,999 days after midnight on the 17th 
10 November 1858 and allows dates to the year 4597. 
The Spatial Co-ordinate Format 

The spatial co-ordinate value consists of three components defined as follows: 

•Altitude: 8 decimal numbers specifying up to 99,999,999 metres. 

•Longitude: 8 decimal numbers specifying East/West 180.00000 degrees (5 
15 decimal places active). 

•Latitude: 8 decimal numbers specifying North/South 90.00000 degrees (5 
decimal places active). 

The Altitude value is expressed as a value in metres from the centre of the earth 
thus allowing altitudes below the sea level. 
20 It should be noted that although spatial co-ordinates are static for most clips, 

this is not true for all cases. Material captured from a moving source such as a camera 
mounted on a vehicle may show changing spatial co-ordinate values. 

Country Code 

The Country code is an abbreviated 4-byte alpha-numeric string according to 
25 the set defined in ISO 3 166. Countries which are not registered can obtain a registered 
alpha-numeric string from the SMPTE Registration Authority. 
Organisation Code 

The Organisation code is an abbreviated 4-byte alpha-numeric string registered 
with SMPTE. Organisation codes have meaning only in relation to their registered 
30 Country code so that Organisation codes can have the same value in different 
countries. 

User Code 
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The User code is a 4-byte alpha-numeric string assigned locally by each 
organisation and is not globally registered. User codes are defined in relation to their 
registered Organisation and Country codes so that User codes may have the same 
value in different organisations and countries. 
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} CLAIMS 

1 . A method of embedding data in material, the method comprising 
combining a representation of the material with a function of the data and a 

scaling factor; wherein 
5 the-scaling factor is generated as a function of a . trial decoding of the material, 

the trial decoding comprising processing the material to recover data therefrom. 

2. A method according to claim 1, wherein the step of combining involves 
modulating a pseudo random symbol sequence with the said data and the trial 

— s j r Q decoding compris e s the step of correl ating a ps e u d o r and om s ymbol seque n ce with the 

said representation of the material, and the scaling factor is generated in dependence 
on the result of the correlation. 

3. A method according to claim 1 or 2, further comprising: 

1 5 combining, as a trial, a representation of the material with a function of the data 

and a trial scaling factor; and 

performing, as a trial, a predetermined process on the combined material and 

data; 

wherein the scaling factor is generated as a function of a trial decoding of the 
20 processed combined material and data. 

4. A method of embedding data bits in material, the method comprising 
the steps of: 

producing transform coefficients Ci representing a transform of the material; 
25 producing a pseudo random symbol stream having L symbols Pi of values +1 

and -1 ; 

calculating a correlation S=E Ci.Pi, for i=l to i=L; and 

calculating modified coefficient values Ci' = Ci + a * Pi, where a is calculated 
dependent on S and the value of the data bit to be embedded in the coefficient. 

30 

5. A method according to claim 4 wherein 
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ct= (<x'+ offset) 

where a' + offset is a function ot the data bit to be embedded in the coefficient, 
a' =0 if S is positive and the data to be concealed is a symbol of a first value, 
a' =0 if S is negative and the data to be concealed is a symbol of a second 
value, and 

otherwise a' is a function of S such that I Ci'.Pi has the correct sign to 
represent the symbol to be encoded. 

6. A method according to claim 5, wherein the first value is "1" and the 
second value is "0" 

7. A method according to claim 4, 5 or 6 wherein the said function of S is 
a'=-S/(L-l)or a' = - S/L. 

15 8. A method according to claim 4, 5, 6 or 7, wherein the magnitude of the 

offset is greater than or equal to zero. 

9. A method according to claim 8, wherein the magnitude of the offset is 

one. 

20 

10. A method according to any preceding claim, wherein the said 
coefficients are coefficients of a wavelet transform of the material. 

11. A method according to any preceding claim wherein the data to be 
25 embedded includes a UMID. 

12. A method of removing data from material, which data has been 
embedded by the method of any one of claims 4 to 11, the method comprising the 
steps of: 

30 calculating the correlation S'=2 Ci'*Pi for i=l to L where Pi are the bits of the 

PRSS and have values +1 and -1; 
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calculating a r as a function of S'; and 

calculating Ci=Ci'-a r .Pi to recover the unmodified coefficients Ci. 

13. A method according to claim 12, wherein <x r =S7(L-l) or cc r =( s yL)- 

5 

14. A method according to claim 12 or 13, further comprising deriving the 
symbols of the concealed data from S\ where if S' is positive a symbol is of the first 
value and if S' is negative a symbol is of the second value. 

10 15. A method according to claim 12, 13 or 14, comprising the step of 

generating and synchronizing a reference pseudo random symbol sequence with the 
pseudo random symbol sequence of the embedded data. 

16. A computer program product arranged to carry out the method of any 
1 5 preceding claim when run on a computer. 

17. Apparatus for embedding data in material, the apparatus comprising: 

a combiner for combining a representation of the material with a function of 
the data and a scaling factor; 
20 a decoder for decoding, as a trial, the material to recover data therefrom; and 

a generator for generating the scaling factor as a function of the trial decoding 
of the material. 

1 8. Apparatus according to claim 17, wherein: 

25 the combiner includes a modulator for modulating a pseudo random symbol 

sequence with the said data; 

the trial decoder comprises a correlator for correlating the pseudo random 
binary sequence with the said material; and 

the scaling factor generator is arranged to generate the scaling factor in 
30 dependence on the result of the correlation. 
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19. Apparatus according to claim 17 or 18, wherein the combiner is 
arranged to combine, as a trial, a representation of the material with a function of the 
data and a trial scaling factor; 

and further comprising: 
5 a processor for performing, as a trial, a predetermined process on the combined 

material and data; 

wherein the scaling factor is generated as a function of a trial processing of the 
processed combined material and data so as to recover the said data. 

1 0 20. Apparatus for embedding data in material, the apparatus comprising: 

a transformer for producing transform coefficients Ci representing a transform 
of the material; 

a generator for producing a pseudo random symbol stream having L bits Pi; 
an input for receiving symbol representing the data to be concealed; and 
1 5 a data embedder arranged to 

calculate a correlation S=Z Ci*Pi, for i=l to i=L and 

calculate modified coefficient values Ci' = Ci + a * Pi, where a is calculated 
dependent on S and the value of the data bit to be embedded in the coefficient. 

20 21 . Apparatus according to claim 20 wherein 

a= (a'+ offset) 

. m where a' + offset is a function ot the data bit to be embedded in the coefficient, 
and the apparatus is arranged to 

calculate modified coefficient values Ci' = Ci + (ct'+ offset)*Pi where 
25 ot'=0 if S is positive and the data to be concealed is a bit of a first value, 

a' =0 if S is negative and the data to be concealed is a bit of a second value, 

and 

otherwise a' is a function of S such that £ Ci\Pi has the correct sign to 
represent the bit to be encoded. 
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) 22. Apparatus according to claim 21 , wherein the first value is "1" and the 

second value is "0". 

23. Apparatus according to claim 21 or 22 wherein a=-S/(L-l) or -S/L 

5 

24. Apparatus according to claim 20, 21, 22 or 23, wherein the offset is 
greater than or equal to zero. 

25. Apparatus according to claim 25 wherein the offset = 1. 

10 

26. Apparatus according to ant one of claims 20 to 25, wherein the 
coefficients are coefficients of a wavelet transform of the material. 

27. Apparatus according to any one of claims 19 to 24 comprising a 
1 5 generator for generating a UMID as the said data to be concealed. 

28. Apparatus for removing data from material, which data has been 
concealed by the apparatus of any one of claims 20 to 27, the apparatus comprising: 

a generator for generating a PRSS; and 
20 a calculator for calculating 

the correlation S'=2 Ci'.Pi for i= 1 to L where Pi are the bits of the PRSS, 
a value ct r dependent on S', and 

a coefficient value Ci=Ci'-a r .Pi to recover the unmodified coefficients Ci. 

25 29. Apparatus according to claim 26, wherein ct r =S/(L-l) or (S/L). 

30. Apparatus according to claim 28 or 29, further comprising a decoder for 
deriving the bits of the concealed data from the correlation value S', where if S' is 
positive a bit of the data has a first value and if S' is negative a bit of the data has a 
30 second value. 
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31 Apparatus according to claim 28, 29 or 30, comprising a synchronizer 
for synchronizing the generated PRSS with the PRSS of the embedded data. 

33. A method of embedding data in material substantially as hereinbefore 
5 described with reference to anyone of Figures 1 to 3 and 6 to 10 of the accompanying 
drawings. 

33. Apparatus for embedding data in material substantially as hereinbefore 
described with reference to anyone of Figures 1 to 3 and 6 to 10 of the accompanying 
10 drawings. 



34. A method of removing 
hereinbefore described with reference 
drawings. 

15 

35. Apparatus for removing 
hereinbefore described with reference 
drawings. 



data embedded in material substantially as 
to Figures 4 and 5 of the accompanying 

data embedded in material substantially as 
to Figures 4 and 5 of the accompanying 



1-00-81, P/ 1003 1GB 



26 



ABSTRACT 



EMBEDDING DATA IN MATERIAL 



10 



15 



A spatial domain image I produced by a source 1 is combined with watermark 
data Ri to produce a spatial domain watermarked image I\ The watermarked image is 
produced by an embedder 3 according to the equation 



where Ci and Ci' are wavelet transform coefficients of the image, and a is a 
scaling factor, a is chosen so that the watermark is imperceptible in the image and to 
resist removal of the watermark by unauthorised processing. It is desirable that a has 
the smallest value which achieves that. If a is too big the watermark is perceptible in 
the image; if it is too small the mark may not survive processing of the image. 

a is determined from a trial decoding of the image I in a decoder 4. The 
decoding is that which would be used to decode the watermarked image V. A value a' 
is produced by a calculator S3-S8, to which an offset value is added by an adder S9 to 
produce a. This produces values of a over the image, which are used to scale the data 
Ri so as to conceal the data. An image is one example of material to which the 
invention is applicable. 



Ci'=Ci + a. Ri 
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[Figure 1] 
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